WIP aarch64 sandbox (configurable sandbox+toolchain) #78

mahaase · 2021-03-01T19:39:09Z

hey guys, i need help! :)

goal is to bring up a configurable sandbox+toolchain.
i want to setup an aarch64 sandbox incl. toolchain.
many stuff is working out of the box, including the aarch64 sandbox, if qemu-aarch64-static is installed.

NOTE: u should use qemu5. older versions have very bad performance! normally u should build it urself. default version of ubuntu18 lts is 2.11... sooooo far behind.

./dev/build/libs/compat/glibc-dev/1/build.sh shell
bash-4.3$ uname -a
Linux bob 4.15.0-135-generic #139-Ubuntu SMP Mon Jan 18 17:38:24 UTC 2021 aarch64 GNU/Linux

there was nothing special i had to do.

general: x86_64 host system -> cross-compiling am aarch64 sandbox -> than we are on a host-aarch64 system, with which we have to go forward to build the project.

i have done already some fixes to get some module buildable for cross-compiling.

this one is blocking the TODO marked stuff: BobBuildTool/bob#399

what do u think about this?

br.

mahaase · 2021-03-01T19:40:12Z

recipes/libs/expat.yaml

@@ -1,14 +1,15 @@
 inherit: [autotools]

 metaEnvironment:
-   PKG_VERSION: "2.1.0"
+   PKG_VERSION: "2.2.10"


includes the support for aarch64 targets.

Ferruck

As always, I'm more than confused when it comes to this host-toolchain vs. target-toolchain stuff. From my perspective it looks good, but I'd like to wait for @jkloetzke's opinion.

Ferruck · 2021-03-02T11:27:59Z

recipes/libs/compat/glibc.yaml

@@ -36,6 +36,7 @@ buildScript: |
            --with-headers=$2/usr/include \
            --enable-obsolete-rpc \
            --libdir=/usr/lib \
+            --disable-werror \


Why is this needed? I'd rather add a patch than disabling werror altogether...

while cross-compiling for aarch64 i had an error: -Werror=missing-attributes
with --disable-werror i was able to build it successfully

mahaase · 2021-03-02T16:31:39Z

i guess this is a bad idea, that i try this xD
the modules seams not ready for aarch64.

first i had some easy problems in modules like expat or libcap. than glibc, i searched hours for the patches.
now the gcc-cross breaks :-/

but the aarch64-sandbox itself works :)
did u think it is possible to get this up?

makes it sens to build a BASEMENT_HOST_COMPAT_TOOLCHAIN if we are inside the sandbox?
everything necessary was built for the sandbox and should be available?!
maybe in this case, i could deactivate this?

jkloetzke · 2021-03-02T19:36:55Z

As always, I'm more than confused when it comes to this host-toolchain vs. target-toolchain stuff. From my perspective it looks good, but I'd like to wait for @jkloetzke's opinion.

Actually it is quite simple: host-toolchain builds for the system where Bob is running on and target-toolchain for, well, the target system. If you are not cross compiling both are identical. In the cross-compiling use case they differ.

but the aarch64-sandbox itself works :)
did u think it is possible to get this up?

I could imagine that it's possible but it would be good to define precisely what you want to accomplish. If I understand correctly then you want to:

create a sandbox that has a foreign architecture
put a statically linked qemu inside the sandbox to translate from the foreign architecture to the host
have some kind of native compiler in the sandbox

Is this correct?

makes it sens to build a BASEMENT_HOST_COMPAT_TOOLCHAIN if we are inside the sandbox?

Depends on you use case where you want to execute Bob. Does the sandbox run on the target? Will it be the same glibc? Or does the toolchain need to run on other Distros too?

mahaase · 2021-03-03T07:59:20Z

put a statically linked qemu inside the sandbox to translate from the foreign architecture to the host

how ever this is not necessary. for me, it works out of the box (with the tool installed on my machine).
if u use this branch and build the sandbox successfully, u can still use it, e.g. by calling the build.sh shell, et voila u are inside a aarch64 sandbox system.

my use-cases i want to bring into the bob:
(+) x86_64 native builds (windows + linux)
(+) x86_64 cross builds (to newer gcc + glibc) with e.g. a docker image in the end to execute the stuff (windows + linux)
(+) x86_64 to aarch64 cross build (our targets are nvidia agx embedded systems) (linux)
(-) aarch64 native build: (linux)
aarch64 sandbox (e.g. on x86_64 jenkins to create artifafcts, which can be used by native nvidia targets)
on the aarch64 nvidia target itself

all with and without cuda (windows + linux)

Does the sandbox run on the target? Will it be the same glibc? Or does the toolchain need to run on other Distros too?

that's the point! we need the host-compat-toolchain because embedded target != sandbox. but we call bob on host (with aarch64 sandbox) and on target without sandbox!

but the glibc problem is a general one! ALL our linux machines (embedded and pc) are ubuntu 18 lts. the current basement target toolchain is still to new for us.

bionic: glibc 2.27; gcc 3.3 to 8; cuda 10.2 (which also supports just gcc until v8)

if the sandbox will be cross-compiled like here, what is the correct way? Also cross-compiling the bootstrap-sandbox, or just the sandbox?

mahaase · 2021-03-03T15:33:09Z

if i get this:

rpc_clntout.c:32:19: fatal error: stdio.h: No such file or directory
compilation terminated.
rpc_hout.c:36:19: fatal error: stdio.h: No such file or directory
compilation terminated.
rpc_sample.c:37:19: fatal error: stdio.h: No such file or directory
compilation terminated.
rpc_svcout.c:36:19: fatal error: stdio.h: No such file or directory
rpc_util.c:36:19: fatal error: stdio.h: No such file or directory

while building: cmake::greeter-cross-aarch64/devel::host-compat-toolchain/devel::compat::cross-toolchain/devel::compat::gcc-cross/libs::compat::glibc-dev

than there is an error in the sandbox environment, isn't it? we missed the installation of the headers to /usr/include...
~~for a x86_64 sandbox, does the system use the host headers?~~ (why this doesn't happen on x86_64...?)

libs::compat::glibc-dev shall be built with devel::compat::gcc-cross-bare which uses --with-sysroot=/does/not/exist?!

jkloetzke · 2021-03-04T19:43:51Z

I'm looking into it but it will take some time.

bionic: glibc 2.27; gcc 3.3 to 8; cuda 10.2 (which also supports just gcc until v8)

I hope gcc 3.3 is a typo. Ubuntu bionic should come with gcc 7.

if the sandbox will be cross-compiled like here, what is the correct way? Also cross-compiling the bootstrap-sandbox, or just the sandbox?

I think the sandbox cross compiling is correct. The bootstrap-sandbox is just an extra layer to make the real sandbox build reproducible. Therefore it should stay in the native architecture.

mahaase · 2021-03-05T07:44:53Z

I hope gcc 3.3 is a typo. Ubuntu bionic should come with gcc 7.

there is a range of installable gcc verions pro ubuntu release. for 18lts there are gcc 4.8 to 8, but indeed not 3.3! default is 7!

I pushed my last ideas, that the sandbox provides its information into AUTOCONF_HOST and AUTOCONF_BUILD.

I'm looking into it but it will take some time.

no pressure! every help is welcome! :)

Ferruck · 2021-03-05T07:52:24Z

recipes/libs/compat/glibc/0003-improve-ifunc-check-BZ-25506.diff.save

+  LC_ALL=C $READELF -Wr conftest | grep -q 'IRELATIVE\|R_SPARC_JMP_IREL' && {
+     libc_cv_ld_gnu_indirect_function=yes
+   }
+ fi


Have you taken this from https://sourceware.org/pipermail/glibc-cvs/2020q1/000002.html ?

If yes, please add the patch with the original author, commit message and a link to the original source (at least in your commit message) included, otherwise this might infringe copyrights...

thanks for that information, but i guess this patch was not needed, i removed it.

jkloetzke · 2021-05-20T19:20:01Z

I think the last changes (--with-linker-hash-style=gnu) should have gone to #84 instead, right?

mahaase · 2021-05-21T05:59:14Z

shit i pushed the merged changes in here... i have to fix this. thanks!

mahaase · 2021-06-30T09:04:53Z

basement/recipes/devel/compat/gcc.yaml

Line 137 in fe037ed

--with-sysroot=/usr/sysroots/${AUTOCONF_TARGET} \

i do not understand. in my opinion this line is not correct. this path will not exists. to which location shall this reference?
to the location of the gcc-cross or to the build-dir of cross-toolchain or something else?

maybe the same like used in gcc-native

basement/recipes/devel/compat/gcc.yaml

Line 155 in fe037ed

TARGET_SYSROOT="${GCC_PREFIX:-/usr}/sysroots/${AUTOCONF_HOST}"

would work?

jkloetzke · 2021-06-30T21:18:28Z

basement/recipes/devel/compat/gcc.yaml

Line 137 in fe037ed

--with-sysroot=/usr/sysroots/${AUTOCONF_TARGET} \

i do not understand. in my opinion this line is not correct. this path will not exists. to which location shall this reference?
to the location of the gcc-cross or to the build-dir of cross-toolchain or something else?

Even though it looks like an absolute path it will still be implicitly relocated by gcc. I don't have the documentation at hand but if the configured sysroot is below the configured install prefix (which is /usr) it will be considered for relocation too. In the end, gcc will look for the sysroot relative to its executable.

The sysroot is actually created in the cross-toolchain recipe when the gcc- and libc-target-files are copied into the sysroot at this location.

mahaase · 2021-07-13T21:47:53Z

i guess, with the last changes the generic sandbox does work. (aarch64 sandbox)
still some retests necessary.

imo the switch from aarch64-linux-gnu to aarch64-cross-linux-gnu was an important step to bring this up without missing headerfile errors.

daxcore · 2021-07-17T14:12:00Z

uhh. still looks fine for me. i also tested the cmake::greeter-cross-aarch64 recipe. a aarch64 sandbox with x86_64 target compiler. still works. :)

…te is an option

…rameters

…te is an option

…rch64 toolchain

.FIXES BobBuildTool#142

mahaase · 2022-06-03T12:41:27Z

the most problems already fixed. there are just some adaptions left. hopefully we can bring this feature to master soon 😃

mahaase · 2022-06-15T13:48:00Z

some important information i will add to the commit message finally:

the "generic sandbox" feature is also working with WSL2 (tested with >= Ubuntu 18 LTS)

sudo apt install qemu binfmt-support qemu-user-static
install a newer version of qemu-user-static (>=5) manually
- https://packages.ubuntu.com/kinetic/amd64/qemu-user-static/download
- sudo dpkg -i <filename>.deb
workaround for WSL2 Ubuntu 18 LTS only (not needed on WSL2 Ubuntu 20 LTS or native Ubuntu >= 18 LTS):
- sudo update-binfmts --unimport
- sudo update-binfmts --import
sudo /etc/init.d/binfmt-support (re-)start or if available: sudo systemctl (re-)start binfmt-support.service
important, check that update-binfmts --display shows interpreter = /usr/libexec/qemu-binfmt/* instead of /usr/bin/qemu-*-static directly

mahaase · 2022-06-16T12:49:49Z

close. follow up here: #154

mahaase commented Mar 1, 2021

View reviewed changes

mahaase force-pushed the feature/aarch64Sandbox branch from dbb3668 to 0a932d3 Compare March 1, 2021 19:41

mahaase mentioned this pull request Mar 1, 2021

hostArch/ARCH is ambiguous #69

Closed

Ferruck reviewed Mar 2, 2021

View reviewed changes

mahaase force-pushed the feature/aarch64Sandbox branch from d6b4807 to f169705 Compare March 2, 2021 15:24

Ferruck reviewed Mar 5, 2021

View reviewed changes

mahaase force-pushed the feature/aarch64Sandbox branch from b6c25d9 to 341de9c Compare May 21, 2021 06:24

mahaase force-pushed the feature/aarch64Sandbox branch 3 times, most recently from fd7690f to 7742581 Compare July 13, 2021 21:40

mahaase force-pushed the feature/aarch64Sandbox branch 2 times, most recently from 497e066 to 9f785f6 Compare July 26, 2021 10:35

mahaase force-pushed the feature/aarch64Sandbox branch from 4f999a0 to d440a34 Compare February 17, 2022 08:24

mahaase force-pushed the feature/aarch64Sandbox branch from d440a34 to 8e7f419 Compare March 10, 2022 16:07

mahaase added 4 commits June 3, 2022 08:52

prototype: aarch64 sandbox (configurable sandbox+toolchain)

e152913

classes/sandbox-toolchain: added ARCH if SANDBOX_ARCH is set

ab5e9e1

libs/compat/glibc: disabled werror

eef9339

libs/compat/glibc: fixes for aarch64, maybe included in 2.27, if upda…

66d53b0

…te is an option

mahaase added 21 commits June 3, 2022 08:52

fixed the usage of a cross-built sandbox

5ff1113

sandbox: provides AUTOCONF vars of itself

438ce99

test: added a specific test-target for aarch64

f406f7a

libs/glibc: added another patch for aarch64 support

49f84e7

devel/sandbox: adapt after environment var fix of bob

50be7ac

libs/compat/glibc: removed unnecessary patch

c532c9c

recipes/devel/gcc: added --with-linker-hash-style=gnu to gcc build pa…

3ec2468

…rameters

devel/sandbox: small fix, set default values

0c2744c

prototype: aarch64 sandbox (configurable sandbox+toolchain)

16bde15

libs/compat/glibc: fixes for aarch64, maybe included in 2.27, if upda…

ac84124

…te is an option

fixed the usage of a cross-built sandbox

41e4dd0

test: added a specific test-target for aarch64

1b8a41a

libs/glibc: added another patch for aarch64 support

5378e6e

libs/compat/glibc: removed unnecessary patch

51cbbff

prototype: aarch64 sandbox (configurable sandbox+toolchain)

57858f3

devel/cross-toolchain: autoconf_target aarch64-cross-linux-gnu for aa…

6dbfbff

…rch64 toolchain

classes/cmake added sysroot to find_root_paths

83dcb5d

classes/cmake: add sysroot just for gcc/g++

8cd54bc

classes/cmake: fixed bash if syntax

671e671

cleanup and rebase with master

213db25

fixed wrong rebasement fixes

549193b

mahaase force-pushed the feature/aarch64Sandbox branch from 8e7f419 to 549193b Compare June 3, 2022 08:42

mahaase added 2 commits June 3, 2022 13:44

devel/git: add cross-compilation support

1b3f8b1

.FIXES BobBuildTool#142

.reintegrated upstream fixes

bd0b265

mahaase changed the title ~~prototype: aarch64 sandbox (configurable sandbox+toolchain)~~ WIP aarch64 sandbox (configurable sandbox+toolchain) Jun 3, 2022

mahaase closed this Jun 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP aarch64 sandbox (configurable sandbox+toolchain) #78

WIP aarch64 sandbox (configurable sandbox+toolchain) #78

mahaase commented Mar 1, 2021 •

edited

Loading

mahaase Mar 1, 2021

Ferruck left a comment

Ferruck Mar 2, 2021

mahaase Mar 2, 2021

mahaase commented Mar 2, 2021 •

edited

Loading

jkloetzke commented Mar 2, 2021

mahaase commented Mar 3, 2021 •

edited

Loading

mahaase commented Mar 3, 2021 •

edited

Loading

jkloetzke commented Mar 4, 2021

mahaase commented Mar 5, 2021

Ferruck Mar 5, 2021

mahaase Mar 8, 2021

jkloetzke commented May 20, 2021

mahaase commented May 21, 2021

mahaase commented Jun 30, 2021 •

edited

Loading

jkloetzke commented Jun 30, 2021

mahaase commented Jul 13, 2021

daxcore commented Jul 17, 2021

mahaase commented Jun 3, 2022

mahaase commented Jun 15, 2022 •

edited

Loading

mahaase commented Jun 16, 2022

WIP aarch64 sandbox (configurable sandbox+toolchain) #78

WIP aarch64 sandbox (configurable sandbox+toolchain) #78

Conversation

mahaase commented Mar 1, 2021 • edited Loading

mahaase Mar 1, 2021

Choose a reason for hiding this comment

Ferruck left a comment

Choose a reason for hiding this comment

Ferruck Mar 2, 2021

Choose a reason for hiding this comment

mahaase Mar 2, 2021

Choose a reason for hiding this comment

mahaase commented Mar 2, 2021 • edited Loading

jkloetzke commented Mar 2, 2021

mahaase commented Mar 3, 2021 • edited Loading

mahaase commented Mar 3, 2021 • edited Loading

jkloetzke commented Mar 4, 2021

mahaase commented Mar 5, 2021

Ferruck Mar 5, 2021

Choose a reason for hiding this comment

mahaase Mar 8, 2021

Choose a reason for hiding this comment

jkloetzke commented May 20, 2021

mahaase commented May 21, 2021

mahaase commented Jun 30, 2021 • edited Loading

jkloetzke commented Jun 30, 2021

mahaase commented Jul 13, 2021

daxcore commented Jul 17, 2021

mahaase commented Jun 3, 2022

mahaase commented Jun 15, 2022 • edited Loading

mahaase commented Jun 16, 2022

mahaase commented Mar 1, 2021 •

edited

Loading

mahaase commented Mar 2, 2021 •

edited

Loading

mahaase commented Mar 3, 2021 •

edited

Loading

mahaase commented Mar 3, 2021 •

edited

Loading

mahaase commented Jun 30, 2021 •

edited

Loading

mahaase commented Jun 15, 2022 •

edited

Loading